{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/5-Fine%20Tuning/5_2_LoRA_Tuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j9Y40X0WCY_H"
      },
      "source": [
        "<div>\n",
        "    <h1>Large Language Models Projects</a></h1>\n",
        "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
        "    <h2>5.2-Efficient Fine-Tuning with LoRA.</h2>\n",
        "    <h3></h3>\n",
        "</div>\n",
        "\n",
        "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
        "__________\n",
        "Models: bigscience/bloomz-560m / bigscience/bloom-1b1 / meta-llama/Llama-3.2-1B\n",
        "\n",
        "Colab Environment: GPU T4. You can use a better GPU if you want to execute the notebook faster.\n",
        "\n",
        "Keys:\n",
        "* Fine-Tunning.\n",
        "* LoRA.\n",
        "\n",
        "Related Article: [Efficient Fine-Tuning with LoRA. Optimal training for Large Language Models.](https://levelup.gitconnected.com/efficient-fine-tuning-with-lora-optimal-training-for-large-language-models-266b63c973ca?sk=85d7b5d78e64e568faedfe07a35f81bd)\n",
        "_________\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bCinY2l-upqy"
      },
      "source": [
        "# LoRA Tuning\n",
        "\n",
        "In this notebook I'm introducing how to apply LoRA Tuning with the PEFT library to a pre-trained model.\n",
        "\n",
        "For a complete list of Models compatible with PEFT refer to their [documentation](https://huggingface.co/docs/peft/main/en/index#supported-methods).\n",
        "\n",
        "A short sample of models families available to be trained with PEFT are: Bloom, Llama, GPT-J, GPT-2, BERT... and more. Hugging Face is working hard to bring more Models to the Library.\n",
        "\n",
        "## Brief introduction to LoRA Tuning.\n",
        "LoRA is a re-parameterization technique. Its operation is simple, complex, and brilliant at the same time. It involves reducing the size of the matrices to be trained by dividing them in such a way that when multiplied, they yield the original matrix.\n",
        "\n",
        "The weights that are modified are those of the reduced matrices, not the original matrix. It's better visualized in an image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cp_7rR4Nx7My"
      },
      "source": [
        "![Matrices_LoRA.webp]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0oSLJkXty-t_"
      },
      "source": [
        "We have an original matrix of 50x50, which means we would have to modify about 2500 parameters. However, as we know, if we multiply two matrices of (2x50) and (50x2), we obtain a 50x50 matrix. Yet, these two matrices are formed by only 100 parameters each. In other words, for the reduced matrices, we need to modify a total of 200 parameters compared to the 2500 of the original matrix. This represents a 92% reduction, and the larger the original matrix, the greater the percentage of savings.\n",
        "\n",
        "In Language Models like GPT-3 or any of the current ones with LoRA, it's possible that we only need to train about 0.02% of the original parameters. This varies for each model. The best part is that the obtained result is very similar to that of full fine-tuning, in some cases, it can even be better."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "620MdVMk7iUS"
      },
      "source": [
        "# Load the PEFT and Datasets Libraries.\n",
        "\n",
        "The PEFT library contains the Hugging Face implementation of differente fine-tuning techniques, like LoRA Tuning.\n",
        "\n",
        "Using the Datasets library we have acces to a huge amount of Datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_UyyuMGnCPjA",
        "outputId": "4a633656-fd60-4259-ecb3-37f356d8d41c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.1/199.1 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m51.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.6/24.6 MB\u001b[0m \u001b[31m31.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m188.7/188.7 MB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m96.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.0/542.0 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m172.0/172.0 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "gcsfs 2025.3.0 requires fsspec==2025.3.0, but you have fsspec 2024.3.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!pip install -q peft==0.10.0\n",
        "!pip install -q datasets==2.19.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VOnJlBY-81Wl"
      },
      "source": [
        "From the transformers library we import the necesary classes to import the model and the tokenizer.\n",
        "\n",
        "Then we can load the Tokenizer and the model.\n",
        "\n",
        "Bloom is one of the smallest and smarter model available to be trained with PEFT Library using Prompt Tuning. You can use either of the models in the Bloom Family, I encorage you to use at least two of them and see the differences.\n",
        "\n",
        "I'm using the smallest one just to spend less time trainig, and avoid memory problems in Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vziwd2UuCYGl"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "#model_name = \"bigscience/bloomz-560m\"\n",
        "#model_name=\"bigscience/bloom-1b1\"\n",
        "#modules = [\"query_key_value\"] #Bloom modules\n",
        "\n",
        "model_name = \"meta-llama/Llama-3.2-1B\"\n",
        "modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"] # Llama Modules\n",
        "\n",
        "device = \"cuda\" #\"mps\" for Silicon chips, \"cuda\" for NVIDIA GPUs, or \"cpu\" for no GPU.\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "foundation_model = AutoModelForCausalLM.from_pretrained(model_name,\n",
        "                                                        device_map = device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qtc1gbK39Hp7"
      },
      "source": [
        "## Inference with the pre-trained model.\n",
        "I'm going to do a test with the pre-trained model without fine-tuning, to see if something changes after the fine-tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "jak6FzpvFTHk"
      },
      "outputs": [],
      "source": [
        "#this function returns the outputs from the model received, and inputs.\n",
        "def get_outputs(model, inputs, max_new_tokens=100):\n",
        "    outputs = model.generate(\n",
        "        input_ids=inputs[\"input_ids\"],\n",
        "        attention_mask=inputs[\"attention_mask\"],\n",
        "        max_new_tokens=max_new_tokens,\n",
        "        repetition_penalty=1.5, #Avoid repetition.\n",
        "        early_stopping=True, #The model can stop before reach the max_length\n",
        "\n",
        "        eos_token_id=tokenizer.eos_token_id\n",
        "    )\n",
        "    return outputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkFqjS459jAa"
      },
      "source": [
        "The dataset used for the fine-tuning contains prompts to be used with Large Language Models.\n",
        "\n",
        "I'm going to request the pre-trained model that acts like a motivational coach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3BAYg7czFYeK",
        "outputId": "ba3c8e4c-2762-4886-bfd7-4860529a629c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The following generation flags are not valid and may be ignored: ['early_stopping']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n",
            "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['I want you to act as a motivational coach. \\xa0You’ll be the one who’s in charge of getting people motivated and excited about their work, so that they can do it better.\\nWhat does this look like? How should I go into action?\\nThere are two ways for me:\\n1)']\n"
          ]
        }
      ],
      "source": [
        "#Inference original model\n",
        "input_sentences = tokenizer(\"I want you to act as a motivational coach. \", return_tensors=\"pt\")\n",
        "foundational_outputs_sentence = get_outputs(foundation_model,\n",
        "                                            input_sentences.to(device),\n",
        "                                            max_new_tokens=50)\n",
        "\n",
        "print(tokenizer.batch_decode(foundational_outputs_sentence, skip_special_tokens=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KQUGY47p9ysI"
      },
      "source": [
        "Not sure if the answer is correct or not, but for sure is not a prompt. We need to train our model if we want that acts like a prompt engineer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL5L_DcR9ggA"
      },
      "source": [
        "# Preparing the Dataset.\n",
        "The Dataset used is:\n",
        "\n",
        "https://huggingface.co/datasets/fka/awesome-chatgpt-prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DyIMQ7IHFbIx"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "dataset = \"fka/awesome-chatgpt-prompts\"\n",
        "\n",
        "#Create the Dataset to create prompts.\n",
        "data = load_dataset(dataset)\n",
        "data = data.map(lambda samples: tokenizer(samples[\"prompt\"]), batched=True)\n",
        "train_sample = data[\"train\"].select(range(50))\n",
        "\n",
        "train_sample = train_sample.remove_columns('act')\n",
        "\n",
        "display(train_sample)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gmlZY3fk_9fm",
        "outputId": "8fdc8cb0-9a58-47b0-b9e7-39a35015653e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'prompt': ['Imagine you are an experienced Ethereum developer tasked with creating a smart contract for a blockchain messenger. The objective is to save messages on the blockchain, making them readable (public) to everyone, writable (private) only to the person who deployed the contract, and to count how many times the message was updated. Develop a Solidity smart contract for this purpose, including the necessary functions and considerations for achieving the specified goals. Please provide the code and any relevant explanations to ensure a clear understanding of the implementation.'], 'input_ids': [[128000, 52157, 499, 527, 459, 10534, 35046, 16131, 51920, 449, 6968, 264, 7941, 5226, 369, 264, 18428, 50596, 13, 578, 16945, 374, 311, 3665, 6743, 389, 279, 18428, 11, 3339, 1124, 34898, 320, 898, 8, 311, 5127, 11, 47005, 320, 2039, 8, 1193, 311, 279, 1732, 889, 27167, 279, 5226, 11, 323, 311, 1797, 1268, 1690, 3115, 279, 1984, 574, 6177, 13, 8000, 264, 22925, 488, 7941, 5226, 369, 420, 7580, 11, 2737, 279, 5995, 5865, 323, 38864, 369, 32145, 279, 5300, 9021, 13, 5321, 3493, 279, 2082, 323, 904, 9959, 41941, 311, 6106, 264, 2867, 8830, 315, 279, 8292, 13]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}\n"
          ]
        }
      ],
      "source": [
        "print(train_sample[:1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVPAJsrUAHiJ"
      },
      "source": [
        "# Fine-Tuning.\n",
        "First is necesary create a LoRA config.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "uCalslQFGL7K"
      },
      "outputs": [],
      "source": [
        "# TARGET_MODULES\n",
        "# https://github.com/huggingface/peft/blob/39ef2546d5d9b8f5f8a7016ec10657887a867041/src/peft/utils/other.py#L220\n",
        "\n",
        "import peft\n",
        "from peft import LoraConfig, get_peft_model, PeftModel\n",
        "\n",
        "\n",
        "lora_config = LoraConfig(\n",
        "    r=8, #As bigger the R bigger the parameters to train.\n",
        "    lora_alpha=16, # A multiplier typically r * 2.\n",
        "    target_modules=modules, #You can obtain a list of target modules in the URL above.\n",
        "    lora_dropout=0.05, #Helps to avoid Overfitting.\n",
        "    bias=\"none\", # Controls whether the bias term is adjusted alongside the model weights during training.\n",
        "    task_type=\"CAUSAL_LM\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LUddynl0B1Ck"
      },
      "source": [
        "The most important parameter is **r**, it defines how many parameters will be trained. As bigger the valuer more parameters are trained, but it means that the model will be able to learn more complicated relations between input and output.\n",
        "\n",
        "Yo can find a list of the **target_modules** available on the [Hugging Face Documentation]( https://github.com/huggingface/peft/blob/39ef2546d5d9b8f5f8a7016ec10657887a867041/src/peft/utils/other.py#L220)\n",
        "\n",
        "**lora_dropout** is like the commom dropout is used to avoid overfitting.\n",
        "\n",
        "**bias** I was hesitating if use *none* or *lora_only*. For text classification the most common value is none, and for chat or question answering, *all* or *lora_only*.\n",
        "\n",
        "**task_type**. Indicates the task the model is beign trained for. In this case, text generation.\n",
        "\n",
        "### Create the PEFT model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BG1zBmhsGQ-h",
        "outputId": "a1edc557-fc33-4cee-ad65-0edbb16b23d3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "trainable params: 5,636,096 || all params: 1,241,450,496 || trainable%: 0.4539928106807088\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "peft_model = get_peft_model(foundation_model, lora_config)\n",
        "print(peft_model.print_trainable_parameters())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2onXUsaw-Ga0"
      },
      "source": [
        "The number of trainable parameters is really small compared with the total number of parameters in the pre-trained model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "HArPQ_lvGUkY"
      },
      "outputs": [],
      "source": [
        "#Create a directory to contain the Model\n",
        "import os\n",
        "working_dir = './'\n",
        "\n",
        "output_directory = os.path.join(working_dir, \"peft_lab_outputs\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rWalmqWm4STo"
      },
      "source": [
        "In the TrainingArgs we inform the number of epochs we want to train, the output directory and the learning_rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "ND0aJ-t6ARqD"
      },
      "outputs": [],
      "source": [
        "#Creating the TrainingArgs\n",
        "import transformers\n",
        "from transformers import TrainingArguments, Trainer\n",
        "training_args = TrainingArguments(\n",
        "    output_dir=output_directory,\n",
        "    auto_find_batch_size=True, # Find a correct batch size that fits the size of Data.\n",
        "    learning_rate= 3e-2, # The Learning Rate is usually higher than in full fine-tuning.\n",
        "    #optim=\"sgd\", #Use only to test a different optimizer\n",
        "    num_train_epochs=2,\n",
        "    use_cpu=False,\n",
        "    report_to=\"none\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgxsV-iy_J_o"
      },
      "source": [
        "Now we can train the model.\n",
        "To train the model we need:\n",
        "\n",
        "\n",
        "*   The PEFT Model.\n",
        "*   The training_args\n",
        "* The Dataset\n",
        "* The result of DataCollator, the Dataset ready to be procesed in blocks.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "id": "z5NYHqBnGZyF",
        "outputId": "aeb3e669-1fda-44f7-cd3f-18c4cd84e6ea"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='14' max='14' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [14/14 00:16, Epoch 2/2]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=14, training_loss=12.348753792898995, metrics={'train_runtime': 18.625, 'train_samples_per_second': 5.369, 'train_steps_per_second': 0.752, 'total_flos': 70554536681472.0, 'train_loss': 12.348753792898995, 'epoch': 2.0})"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ],
      "source": [
        "## Llama configuration\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "tokenizer.padding_side = \"right\"\n",
        "\n",
        "#This cell may take up to 15 minutes to execute.\n",
        "trainer = Trainer(\n",
        "    model=peft_model,\n",
        "    args=training_args,\n",
        "    train_dataset=train_sample,\n",
        "    data_collator=transformers.DataCollatorForLanguageModeling(\n",
        "        tokenizer,\n",
        "        mlm=False)\n",
        ")\n",
        "trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "kEKiFdpDGgOx"
      },
      "outputs": [],
      "source": [
        "#Save the model.\n",
        "peft_model_path = os.path.join(output_directory, f\"lora_model\")\n",
        "\n",
        "trainer.model.save_pretrained(peft_model_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_TAjrSWSe14q",
        "outputId": "24e40bce-c7d9-4e0a-eb35-c27e9806d1fe"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PeftModelForCausalLM(\n",
              "  (base_model): LoraModel(\n",
              "    (model): LlamaForCausalLM(\n",
              "      (model): LlamaModel(\n",
              "        (embed_tokens): Embedding(128256, 2048)\n",
              "        (layers): ModuleList(\n",
              "          (0-15): 16 x LlamaDecoderLayer(\n",
              "            (self_attn): LlamaAttention(\n",
              "              (q_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=2048, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=2048, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (k_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=512, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=512, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (v_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=512, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=512, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (o_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=2048, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=2048, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "            )\n",
              "            (mlp): LlamaMLP(\n",
              "              (gate_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=8192, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=8192, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (up_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=2048, out_features=8192, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=2048, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=8192, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (down_proj): lora.Linear(\n",
              "                (base_layer): Linear(in_features=8192, out_features=2048, bias=False)\n",
              "                (lora_dropout): ModuleDict(\n",
              "                  (default): Dropout(p=0.05, inplace=False)\n",
              "                )\n",
              "                (lora_A): ModuleDict(\n",
              "                  (default): Linear(in_features=8192, out_features=8, bias=False)\n",
              "                )\n",
              "                (lora_B): ModuleDict(\n",
              "                  (default): Linear(in_features=8, out_features=2048, bias=False)\n",
              "                )\n",
              "                (lora_embedding_A): ParameterDict()\n",
              "                (lora_embedding_B): ParameterDict()\n",
              "              )\n",
              "              (act_fn): SiLU()\n",
              "            )\n",
              "            (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "            (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "          )\n",
              "        )\n",
              "        (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "        (rotary_emb): LlamaRotaryEmbedding()\n",
              "      )\n",
              "      (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
              "    )\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "#Load the Model.\n",
        "loaded_model = PeftModel.from_pretrained(foundation_model,\n",
        "                                        peft_model_path,\n",
        "                                        is_trainable=False)\n",
        "loaded_model.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dK--YFPR6OxH"
      },
      "source": [
        "## Inference the fine-tuned model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I_27uvJudf03",
        "outputId": "d03795b9-efab-4cb4-d5c2-dc13916cc698"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The following generation flags are not valid and may be ignored: ['early_stopping']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n",
            "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['I want you to act as a motivational coach.  express touch first better information plan my form part practice their need make other of use type is book about means become with what over reason My like list way reasons watch second personal help talk making creating the experience can into be ready places show trying glass break create']\n"
          ]
        }
      ],
      "source": [
        "input_sentences = tokenizer(\"I want you to act as a motivational coach. \", return_tensors=\"pt\")\n",
        "finetuned_outputs_sentence = get_outputs(loaded_model,\n",
        "                                          input_sentences.to(device),\n",
        "                                          max_new_tokens=50)\n",
        "\n",
        "print(tokenizer.batch_decode(finetuned_outputs_sentence, skip_special_tokens=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCV9JOBG6Ug8"
      },
      "source": [
        "The result is amazing. Let's compare the answer of the pre-trained Model withe the one fine-tuned by us using LoRA:\n",
        "* **Pretrained Model:** *I want you to act as a motivational coach.*  Don't be afraid of being challenged.\n",
        "* **Fine-Tuned Model:** I want you to act as a motivational coach.  I will provide some information about someone\\'s motivation and goals, but it should be your job  in order my first request – \"I need someone who can help me find the best way for myself stay motivated when competing against others.\" My suggestion is “I have\n",
        "\n",
        "As you can see the result is really similar to the samples containmed in the Datased used to fine-tune the Model. And we only trained the Model for 10 epochs and with a really small number of rows.\n",
        "\n",
        "# Continue Learning\n",
        "Please, play with all the variables in the notebook and drive your own experiments and get your conclusions.\n",
        "\n",
        "Try to change the **lora_config** values, maybe you can achieve a better result in less epochs, saving time and money for your company. :-)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tr4Sm32y89Ji"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "authorship_tag": "ABX9TyMZ1slRCJ9ejfiVrHMOwOIS",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}